Method and apparatus for estimating area or volume of object of interest from gastrointestinal images

ABSTRACT

A method and apparatus for estimating or measuring a physical area or physical volume of an object of interest in one or more images captured using an endoscope are disclosed. According to the present method, one or more structured-light images and one or more regular images captured using an imaging apparatus are received. An object of interest in the regular images is determined. Distance information associated with the object of interest with respect to the imaging apparatus is derived from the structured-light images. The physical area size or physical volume size of the object of interest is determined based on the regular images and the distance information. The imaging apparatus can be a capsule endoscope or an insertion endoscope.

CROSS REFERENCE TO RELATED APPLICATIONS

The present invention is a continuation-in-part application of andclaims priority to U.S. Patent Application, U.S. Ser. No. 16/416,266,filed on May 20, 2019, which is a continuation-in-part application ofand claims priority to U.S. Patent Application, U.S. Ser. No.15/669,853, filed on Aug. 4, 2017, now U.S. Pat. No. 10,346,978, issuedon Jul. 9, 2019. The present invention is also related to U.S. PatentApplication, U.S. Ser. No. 14/884,788, filed on Oct. 16, 2015, which isnow a U.S. Patent, U.S. Pat. No. 9,936,151, issued on Apr. 3, 2018. TheU.S. Patent Applications and U.S. Patent are hereby incorporated byreference in their entireties.

FIELD OF THE INVENTION

The present invention relates to the endoscope for capturing images ofhuman gastrointestinal (GI) tract for diagnosis purpose. In particular,the endoscope is enabled to estimate physical area or physical volume ofan object of interest in the GI images based on structured light images.

BACKGROUND AND RELATED ART

Devices for imaging body cavities or passages in vivo are known in theart and include endoscopes and autonomous encapsulated cameras.Endoscopes are flexible or rigid tubes that pass into the body throughan orifice or surgical opening, typically into the esophagus via themouth or into the colon via the rectum. An image is formed at the distalend using a lens and transmitted to the proximal end, outside the body,either by a lens-relay system or by a coherent fiber-optic bundle. Aconceptually similar instrument might record an image electronically atthe distal end, for example using a CCD or CMOS array, and transfer theimage data as an electrical signal to the proximal end through a cable.Endoscopes allow a physician control over the field of view and arewell-accepted diagnostic tools.

Capsule endoscope is an alternative in vivo endoscope developed inrecent years. For capsule endoscope, a camera is housed in a swallowablecapsule, along with a radio transmitter for transmitting data, primarilycomprising images recorded by the digital camera, to a base-stationreceiver or transceiver and data recorder outside the body. The capsulemay also include a radio receiver for receiving instructions or otherdata from a base-station transmitter. Instead of radio-frequencytransmission, lower-frequency electromagnetic signals may be used. Powermay be supplied inductively from an external inductor to an internalinductor within the capsule or from a battery within the capsule.

An autonomous capsule camera system with on-board data storage wasdisclosed in the U.S. Pat. No. 7,983,458, entitled “In Vivo AutonomousCamera with On-Board Data Storage or Digital Wireless Transmission inRegulatory Approved Band,” granted on Jul. 19, 2011. The capsule camerawith on-board storage archives the captured images in on-boardnon-volatile memory. The capsule camera is retrieved upon its exitingfrom the human body. The images stored in the non-volatile memory of theretrieved capsule camera are then accessed through an output port on inthe capsule camera.

When the endoscope is used for imaging the human GI tract, one of theprimary purposes is to identify any possible anomaly. If any anomaly isfound, it is further of interest to determine characteristics of theanomaly, such as the size of the anomaly. For example, the polyp size isan important clinical factor associated with surveillance intervaldecision making for the colonoscopy procedure. Usually, a large polypsize is associated with a higher probability of malignancy. Furthermore,for cancerous tumor, the size will affect the probability oflymphovascular invasion and metastasis, and also impact prognosissubstantially. For example, in a technical paper by Warren et al.,(“Comparison of One-, Two-, and Three-Dimensional Measurements ofChildhood Brain Tumors”, Journal of National Cancer Institute, pp.141-145, Vol. 93, No. 18, Sep. 19, 2001), it shows that the tumorlymphovascular metastasis is more closely related to tumor area orvolume, i.e. multiple dimensional measurement than a dimensionalmeasurement. Similar observation has also be noted by Kikuchi et al.,(“A new staging system based on tumor volume in gastric cancer’,Anticancer Research, pp. 2933-2936, Vol. 21, No. 4B, July-August 2001).

However in the colonoscopy standard procedure, the polyp size is alwaysmeasured by its longest dimension. For example, in a technical articleby Chaptini et al, (“Variation in polyp size estimation amongendoscopists and impact on surveillance intervals”, GastrointestinalEndoscopy, pp. 652-659, Volume 80, No. 4: 2014), the polyp size isdetermined by measuring the size of the open forceps from the printedphotograph or images displayed on a display device. Similar sizemeasuring technique has also been mentioned by Plumb et al., (“Terminaldigit preference biases polyp size measurements at endoscopy, computedtomographic colonography, and histopathology”, Endoscopy, pp. 899-908,Vol. 48, October 2016).

It is desirable to develop techniques that can easily measure orestimate the physical area or physical volume of an object of interest.

BRIEF SUMMARY OF THE INVENTION

A method for estimating a physical length, physical area or physicalvolume of an object of interest in a regular image captured using anendoscope is disclosed. According to this method, one or morestructured-light images are received, where the structured-light imagesare captured using the imaging apparatus by projecting structured lightonto a body lumen when the imaging apparatus is in the body lumen. Oneor more regular images are received, where the regular images arecaptured using the imaging apparatus by projecting non-structured lightonto the body lumen when the imaging apparatus is inside the body lumen.The object of interest in the regular images is determined, where theobject of interest corresponds to a target object on a wall of the bodylumen. The distance information associated with the object of interestwith respect to the imaging apparatus is derived from thestructured-light images. The physical area size or physical volume sizeof the object of interest is determined based on the regular images andthe distance information. The imaging apparatus can be a capsuleendoscope or an insertion endoscope.

The steps of determining the physical area size or the physical volumesize of the object of interest may comprise: determining a firstdirection aligned with a longest straight line from one end of theobject of interest to another end of the object of interest; estimatinga first physical dimension of the object of interest based on thelongest straight line measured from said one or more regular imagesscaled by a magnification factor associated with optical configurationof an image sensor of the imaging apparatus; and estimating a secondphysical dimension of the object of interest based on a second measuredlongest length of the object of interest in a second direction scaled bythe magnification factor, wherein the second direction is perpendicularto the first direction. The physical area size of the object of interestis proportional to a product of the first physical dimension of theobject of interest and the second physical dimension of the object ofinterest.

In one embodiment, the object of interest can be determined by outlininga boundary of the object of interest by a user or an artificialintelligence process or jointly by the user and the artificialintelligence process. The step of outlining the boundary of the objectof interest can be performed by the user using an input device toindicate the boundary of the object of interest in said one or moreregular images displayed on a display device. If the object of interestcrosses the boundary of a current regular image into one or moreneighboring regular images, the current regular image and said one ormore neighboring regular images are stitched prior to said determiningthe physical area size or the physical volume size of the object ofinterest. The object of interest can be outlined in the current regularimage and said one or more neighboring regular images to assiststitching the current regular image and said one or more neighboringimages.

The step of determining the object of interest may comprise outlining aboundary of the object of interest automatically using image processing.The physical area size or the physical volume size of the object ofinterest can be determined automatically by using computer executablecodes executed on a computing device. The object of interest maycorrespond to a lesion, pedunculated polyp, sessile serrated polyp, flatlesion or an infected area by Crohn's disease.

In one embodiment, a score board can be generated for an anomaly as anindication of condition of the anomaly by using an aggregate numbercorresponding to an average area or percentage of one or more targetobjects of interest in one or more first regular images representing asection of GI (gastrointestinal) tract, where the target objects ofinterest correspond to one or more infected areas in the GI tractassociated with the anomaly. The section of GI tract may correspond to apart of small bowel, a part of colon or both.

In one embodiment, an index number is generated for an anomaly as anindication of a disease state for the anomaly, where the index number isproportional to a sum of infected areas in said one or more regularimages in a GI (gastrointestinal) section divided by a sum of totalimage areas in said one or more regular images in the GI section.

In one embodiment, the object of interest corresponds to an anomaly andthe physical volume size of the object of interest is derived based on a3D mathematical model for the anomaly and measured 1D or 2D sizeinformation for the object of interest, where the 3D mathematical modelis used to predict depth or volume of the anomaly underneath a mucosalsurface. The 3D mathematical model may correspond to an ellipsoid,ovoid, sphere or disc.

In one embodiment, the physical area size or the physical volume size ofthe object of interest is derived based on the distance information thatis derived at a set of point locations by projecting the structuredlight onto the body lumen. A triangle mesh is generated to cover atopographic surface of the object of interest, where the triangle meshconsists of a set of triangles generated using the set of pointlocations. In one embodiment, an outline identifying the object ofinterest is projected onto the triangle mesh and projected area of atarget set of triangles within the outline projected is determined asthe physical area size of the object of interest. In another embodiment,a vector sum of target triangles of the target set of triangles iscalculated, where each vector area associated with each target trianglecorresponds to a product of area and normal vector of each targettriangle. In still yet another embodiment, a 3D shape model is assumedfor the object of interest and the physical volume size of the object ofinterest is estimated based on a measured 1D or 2D size information forthe object of interest.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates an example of measuring the longest dimension of atumor using a forcipes, where an angle exists between the image planeand the object plane.

FIG. 1B illustrates an example of measuring the longest dimension of anobject of interest using a forcipes to align with the longest dimensionof the tumor as seen from an endoscope image.

FIG. 2 illustrates a simplified example of object dimensiondetermination based on object-camera distance.

FIG. 3A illustrates an example of camera geometry correction using acheckerboard pattern on a cylinder surface at distance Z1 and thecaptured image is corrected or de-warped to compensate the geometrydistortion.

FIG. 3B illustrates an example of camera optic characterization using acheckerboard pattern on a cylinder surface at two distances Z1 and Z2 toderive camera rays.

FIG. 4A illustrates an example of a uniform background on the surface ofa cylinder at two difference distances (Z1 and Z2) from the camera.

FIG. 4B illustrates an example of a captured image by projecting N dotsat distance Z1, where the upper left dot is labelled as q1.

FIG. 4C illustrates an example of a captured image by projecting N dotsat distance Z2, where the upper left dot is labelled as q2.

FIG. 4D illustrates an example of some epipolar lines in the upper-leftportion derived from the captured images in FIG. 4B and FIG. 4C.

FIG. 4E illustrates an example in real scenario, when a target light rayintersects with the tissue surface, the projection of the intersectionpoint falls into this ray's epipolar line as a dot.

FIG. 5 illustrates an example of a triangle mesh formed from thestructured light points.

FIG. 6 illustrates a free hand loop indicating a feature can beprojected onto the triangle mesh.

FIG. 7A illustrates an example that a portion of a triangle inside thefree hand loop is indicated by the hatch-line filled area.

FIG. 7B illustrates an example of computing surface area or crosssection area according to an embodiment of the present invention.

FIG. 8 illustrates an exemplary capsule system with on-board storage.

FIG. 9 illustrates an exemplary flowchart for estimating or measuring aphysical length, physical area or physical volume of an object ofinterest in one or more images captured using an endoscope according toan embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

It will be readily understood that the components of the presentinvention, as generally described and illustrated in the figures herein,may be arranged and designed in a wide variety of differentconfigurations. Thus, the following more detailed description of theembodiments of the systems and methods of the present invention, asrepresented in the figures, is not intended to limit the scope of theinvention, as claimed, but is merely representative of selectedembodiments of the invention. References throughout this specificationto “one embodiment,” “an embodiment,” or similar language mean that aparticular feature, structure, or characteristic described in connectionwith the embodiment may be included in at least one embodiment of thepresent invention. Thus, appearances of the phrases “in one embodiment”or “in an embodiment” in various places throughout this specificationare not necessarily all referring to the same embodiment.

Furthermore, the described features, structures, or characteristics maybe combined in any suitable manner in one or more embodiments. Oneskilled in the relevant art will recognize, however, that the inventioncan be practiced without one or more of the specific details, or withother methods, components, etc. In other instances, well-knownstructures, or operations are not shown or described in detail to avoidobscuring aspects of the invention. The illustrated embodiments of theinvention will be best understood by reference to the drawings, whereinlike parts are designated by like numerals throughout. The followingdescription is intended only by way of example, and simply illustratescertain selected embodiments of apparatus and methods that areconsistent with the invention as claimed herein.

Endoscopes are normally inserted into the human body through a naturalopening such as the mouth or anus. Therefore, endoscopes are preferredto be small sizes so as to be minimally invasive. As mentioned before,endoscopes can be used for diagnosis of human gastrointestinal (GI)tract. The captured image sequence can be viewed to identify anypossible anomaly. For example, polyp is an anomaly that a doctor oftenlooks for during the colonoscopy procedure. The polyp size is animportant clinical factor associated with surveillance interval decisionmaking for the colonoscopy procedure. Moreover Crohn's disease is moreprevalent in the western world. The diagnosis and follow-up of thedisease state include endoscope direct visualization of mucosa surfacein the upper and lower gastrointestinal tracts. However, thegastroenterologist has to insert and maneuver the endoscope through theconvoluted human intestine laboriously in order to observe the mucosasurface and obtain a subject impression of the disease condition, suchas how prevalent the inflammation condition is inside the GI tract.Alternatively, an ingestible capsule endoscope can be used to examinethe GI tract.

If any anomaly is found, it is of interest to identify thecharacteristics of the anomaly. There exists a significant unmet need toassess area or volume of an object of interest in the GI tract. Thisassessment is confounded by a few issues in the current state of the artendoscope technology. The measurement currently is always onedimensional, which uses a device of known size (e.g. a forcipes) toalign and to get close to the polyp. The forcipes is aligned with itslongest dimension to estimate the polyp size by comparison with theforcipes of known size. However the endoscope comprises a flexible cableand when inserted inside a tortuous intestine, the precise maneuvercould not be done with ease and precision. Even if alignment with thelongest dimension is successful, usually there will be an angle betweenthe longest dimension of the tumor and the forcipes. For an ingestiblecapsule, the maneuver is out of question.

Moreover, the lesion might not be on the same plane. For example, in thecase of a flat lesion in colon or the inflammation in the case ofCrohn's disease, this makes the accurate measurement untenable.

FIG. 1A illustrates a cross section view for an example of measuring thelongest dimension of a tumor 110 using a forcipes 120. The distal end130 of the endoscope snake cable 131 is inserted into the GI tract,where lines 140 indicate the mucosal surface. Part of the tumor (112) isabove the mucosal surface 140 and part of the tumor (114) is underneaththe mucosal surface. Furthermore, the distal end 130 has a camera andLED light (not explicitly shown) to capture images. Also, there is anopening at the distal end to allow forcipes 120 to extend out for sizemeasurement. The forcipes is aligned with the longest dimension of thetumor to estimate the tumor size. In FIG. 1A, dash lines 150 indicatethe alignment of the tips of forcipes 120 with the longest dimension ofthe tumor as seen from the endoscope image 180 in FIG. 1B. However, theimage plane 160 and the object of inter plane 170 may not be aligned. InFIG. 1A, there is an angle 0 between the image plane and the object ofinterest plane. The measured longest dimension 162 is shorter than thephysical longest dimension 172 as shown in FIG. 1A.

In order to overcome the deficiencies in the conventional sizemeasurement of an object of interest, an invention of the presentinvention discloses an endoscope system that allows a user to easilymeasure the area or volume of an anomaly. According to the presentinvention, an endoscope, tethered or untethered such as an ingestiblecapsule, with distance measurement capability is disclosed to overcomethe above issues existing in the state of the art in diagnosis.

In PCT Patent Application, Serial No. PCT/US17/15668, filed on Jan. 30,2017, a method to measure 1D size of an object of interest from thecaptured image and distance information has been disclosed. The methodaccording to PCT/US17/15668 relieves the needs for the laboriousprocedure requiring a forcipes in the conventional approach. Accordingto PCT/US17/15668, the true size of an object of interest can beestimated from the measured size in the captured image and a determinedobject distance. In an endoscope, the focal length is known by design.If the distance (also named as object distance in this disclosure)between an object and the camera can be determined, the dimensions of anobject can be determined simply using geometry.

FIG. 2 illustrates a simplified example of object dimensiondetermination based on object-camera distance. In a camera system, theimage sensor is placed at the focal plane 220 behind the lens 210. Thecamera can capture a scene within the field of view extending an anglea. The focal length f is the distance between the lens and the imagesensor. The focal length often is fixed for endoscopic applications andis known by design. However, when a capsule endoscope travels throughthe GI tract, the object distance D varies depending on the location ofthe capsule endoscope and its relative angles with respect to the GIwall being imaged. If the distance D is known, the dimension of anobject can be determined from the captured image by measuring the sizeof the object image in the image. For example, if a flat object 230 withheight H is at distance D from the camera, the object image height H canbe derived from the object image height h in the image according to:

$\begin{matrix}{H = {\left( \frac{D}{f} \right){h.}}} & (1)\end{matrix}$

In the above equation, h is measured from the image, the focal length fis known by design, and the distance D is determined by a selecteddistance measuring means as mentioned above. Accordingly, if thedistance can be determined, the object dimensions can be derived. For anactual object, a single object-camera distance is not adequate toestimate the object size in area or volume accurately. Therefore, it isdesirable to determine object-camera distance at multiple locations. InFIG. 2, object 250 corresponds to a cross section of a 3D object.Locations 252, 254 and 256 corresponds to three locations on the objectsurface. With object-camera distances (z-axis) known at multiplelocations (x and y axis), a 3D model of the object surface can beestablished and the surface are of the object can be estimatedaccordingly. For the locations without the object-camera distance, thedistance information can be interpolated from known object-cameradistances.

According to embodiments of the present invention, the object size inthe image is measured in physical dimension. The image is captureddigitally and the size measurement may be more convenient in terms ofthe number of pixels. Since the physical dimension of image sensorsurface and the optical footprint are known. Also, the number of pixelsis known (e.g. 320×240). Therefore, the object image size in the imagecan be measured in a number of pixels and converted physical objectimage size in the image. For convenience, the ratio D/f is referred asmagnification factor. With the object-camera distance known at alocation (i.e., a 3D location), a small area around the location on thesurface can be assigned with this magnification factor for areaestimation. Furthermore, if a 3D model is derived for the objectsurface, the surface area can be calculated using any known areacalculation technique. For example, a net of polygons for the surface isdetermined and the area of the surface can be calculated as a sum ofareas of individual polygons.

As shown above, the object image size in the image depends on the actualobject size and its distance from the camera. A smaller object at acloser distance may appear to have the same size in the image as alarger object at a farther distance. For example, the object 240, whichis smaller but closer than object 230, appears to have the same heightas object 230 in the image. Therefore, the distance is crucialinformation for determining the object size. Accordingly, the distancemeasuring means disclosed above enables object size determination basedon the images captured using an endoscope.

In U.S. Patent, U.S. Pat. No. 9,936,151, issued to the same assignee asthe present application, a method of capturing one or morestructured-light images and associated regular images are disclosed. Themethod captures a structured-light image and an associated regular imagewith a reduced frame interval in between so that the amount of anymovement between the two images is reduced. Accordingly, the depth ordistance information derived from the structured-light image accordingto U.S. Patent, U.S. Pat. No. 9,936,151 is more correlated with theassociated regular image. However, the present invention is not limitedto the depth/distance derivation based on U.S. Pat. No. 9,936,151. Thepresent invention may use any depth/distance derivation method based onthe structured-light images. An example of object-camera distancederivation is described as follows.

The system can use N light beams to detect the object-camera distance atN locations. However, before the system can detect the distance, somecalibrations have to be performed for the camera system and the lightbeam projector. In order to characterize the camera geometry, some knownpictures can be used as the target. For example, a picture withcheckerboard patterns can be used as the target. In order to imitate theshape of the GI tract, a checkerboard pattern on a surface of a cylinderis used as the target picture. The camera geometry calibration isperformed in a controlled environment. As shown in FIG. 3A, thecheckerboard pattern 310 on a cylinder surface at distance Z1 is used asthe target test image. Image 320 corresponds to the image captured usingthe camera. Due to the close distance in the GI environment and othervarious reasons, the captured image is subject to various distortion, inparticularly the radial distortion. Therefore, the captured image needsto be corrected. Image 330 corresponds to the corrected/dewarped image.Each corner pixel p(x1,y1) on the corrected/dewarped image correspondsto a 3D location P(X1,Y1,Z1) on cylinder 1.

Since we know the cylinder distance, size and location of each square ofthe checkerboard, we can easily calculate the 3D location of each cornerat checkerboard. In other words, given a pixel p1 (332) on image 1, weknow its 3D location P1 (312) on cylinder 1. For any other pixel insidecorners, we can use bilinear interpolation to recover its 3D location.In order to establish a camera geometry model, an embodiment accordingto the present invention captures two pictures at two differentdistances (e.g. Z1 and Z2). For image 2, the checkerboard pattern on thecylinder 340 is displayed at distance Z2 as shown FIG. 3B. An image iscaptured using the camera and the captured image is corrected to image350. The 3D location P2 342 of the checkerboard at distance Z2corresponds to the 3D location P1 312 on the checkerboard at distanceZ1. Now for each image pixel (e.g. same pixel p (332) in image 1 (330)and pixel p (352) in image 2 (350)) in the corrected image plane, weknow a pair of 3D points, {P1, P2}. Each pair of two 3D points defines acamera ray for the corresponding image pixel p. This will replace themodel that uses camera optical center of projection to form a cameraray. Based on our experiments, the capsule system cannot be properlymodeled as a perfect pinhole camera. While the checkerboard pattern isused as an example of test image, other test images with known patternsmay also be used. For example, a group of “dots” with size, color orline type variations at known locations can be used so that individual“cross marks” can be discerned.

The camera system, particularly the camera optical system, for eachcapsule device is subject to manufacturing variations. Therefore, thecamera geometry has to be calibrated individually for each capsuledevice before the capsule camera placed for use. The camera calibrationdata can be stored in a non-volatile memory such as NVRAM (non-volatilerandom access memory).

After camera geometry is calibrated, the system calibrates the projectorgeometry by projecting N light beams to a uniform background (e.g.white/gray background) at two different distances (e.g. Z1 and Z2corresponding to the minimum and maximum intended object distance). Forexample, FIG. 4A illustrates an example of a uniform background (410 and420) on the surface of a cylinder at two difference distances (Z1 andZ2) from the camera. For projecting the light beams, a light source(e.g. a laser or an LED (light emitting diode) can be used to projectmultiple light beams through an optical system to the field of view. Forexample, a light beam is projected to location P1 in the background atdistance Z1 and location P2 in the background at distance Z2. Projectionof P1 and P2 on corrected image plane will be pl and p2 respectively.Correction based on the camera geometry can be applied to the epipolarlines to become straight. Using the step similar to the camera geometrycalibration, the pair P1 and P2 forms a projector ray.

FIG. 4B corresponds to a captured image by projecting N dots at distanceZ1, where the upper left dot is labelled as q1. FIG. 4C corresponds to acaptured image by projecting N dots at distance Z2, where the upper leftdot is labelled as p2. Each corresponding dot pair forms an epipolarline. Some epipolar lines in the upper-left portion are shown in FIG.4D. In practice, the projection system for each capsule camera issubject to manufacturing variation. The projector geometry calibrationis performed individually. The projector geometry calibration data canbe stored inside the capsule camera using non-volatile memory such asNVRAM.

In real scenario, when the same light ray intersects with the tissuesurface, the projection of the intersection point will fall into thisray's epipolar line as a dot (e.g. position q in FIG. 4E). For each dotof the projected beams, we need to detect and identify the dot. Since weknow the projector ray of this epipolar line as well as the camera rayfor the pixel at the center of the dot, we can use standardtriangulation to compute the 3D location at the intersection of abovetwo rays. One such method is called mid-point method. According to thismethod, we first define L1 as the projector ray and L2 as the cameraray. Furthermore, we define d(P, L) as the Euclidean distance betweenray L and a 3D point P. We will find the best P which minimizesd(P,L1)²+d(P,L2)². By this means, we can recover the 3D position for allN dots. (another maybe unnecessary clarification: It is necessary tofind the correct correspondence between the projected beams and thespots on the image. This is done with an assignment algorithm. Someepipolar lines cross, sometimes dots are missing and sometimes there isextra stray light. The assignment algorithm attempts to account for allthese possibilities by choosing the solution that best matches typicalgeometry found in the GI tract).

In order to reduce the required computation, an embodiment of thepresent invention builds a lookup table after the epipolar lines arecalculated. We sample M points along each epipolar line and precomputethe 3D position for each sampled point using above triangulation. Forexample, if an epipolar line corresponds to an object distance between 0cm to 5 cm, we can divide the range into target distances at 0.1 cmincrement (i.e., 0, 0.1, 0.2, . . . , 4.8, 4.9, 5.0 cm) and precomputethe 3D points for target distances within the range. If a dot fallsbetween two sampled points, we use linear interpolation to compute the3D position.

An example of interpolation is illustrates as follows. First, the weightw is computed according to:w=|p−p1|/|p2−p1|.

The corresponding 3D position Z will be computed according to:Z=Z1+w*(Z2−Z1)

In the above, one example of distance derivation is illustrated.However, the present invention is not limited to this particulardistance derivation method.

In one embodiment of the present invention, the boundary of the lesionis outlined first. The method based on this embodiment identifies thefirst longest dimension and measure its length according to theinformation of distance and magnification of the imager. Since theendoscope image plane may not be aligned with the lesion plane well, thedistance information from a multiple point in the field of view may berequired.

For the embodiment above, the present method may further provide a toolto find the second longest dimension in the direction perpendicular tothe direction of the first longest dimension. The 2D measurement of thelesion can be expressed as the product of the first longest dimensionand the second longest dimension that are derived above with the 1Ddimension information as a by-product in the process.

In another embodiment, a method according to this embodiment provides atool to measure the real area accurately by using a variety of areameasuring methods, such as using grids.

In yet another embodiment, a method according to this embodiment allowsa user (e.g. a doctor or a medical professional), an algorithmimplemented in computer/software codes or both to outline the lesion inthe image. For example, a medical professional may draw the outline on atouch screen that displays the GI image being examined. A computer mayrun a program trained by deep learning to automatically draw the outlineof a lesion. In yet another example, a doctor may point to a location onthe lesion in the image and the computer with AI (artificialintelligence) may take over to finish the outlining automatically. Theuse of deep learning or artificial intelligence to perform various imageprocessing tasks (e.g. pattern recognition) is well known in the field.The details are not repeated here.

For the GI image, each image typically covers limited a field of view.Therefore, it may occur that a tumor, especially the Crohn's diseaseinflammation, may cross the image boundaries. The whole tumor may spreadover a number of images. According to one embodiment of the presentinvention, image stitching is performed first and area or volumemeasurement is performed based on the stitched image.

While fully automated area or volume measure of an object of interest isfeasible, the system may perform faster or more reliably with somedegree of human assistance. For example, the stitching across multipleimages may be assisted by the outlining boundaries of the lesion, whichcan be designated as a “feature”. The method then estimates the size orvolume of the “feature”.

During the process of estimating the area or volume based on stitchedimage, a score board can be kept by using an aggregate number. Theaggregate number may correspond to an average area or percentage ofinfected area per image. The aggregate number may also be calculated foreach area (e.g. aggregated areas of all images) calculated through asection of the GI tract, such as the small bowel, ileum, colon, etc.Therefore, the disease state can be indicated by an index numberproportional to the sum of infected areas in the total images in a GIsection divided by the sum of image areas of images in the section.

The 3D size (i.e., the volume) may be calculated based on a mathematicalmodel for each type of lesion, pedunculated polyp, sessile serratedpolyp, flat lesion, etc. This model may predict the depth of the lesionunder the surface, which is useful in the assessment of probability ofmetastasis.

According to another embodiment of the present invention, structuredlight is used to estimate the 3D size. For example, a number of pointlocations in 3D can be obtained by using structured light, where thestructured light ray intersects with the walls of the lumen. Anapproximate continuous depth map can be formed by connecting the knownpoints together in a triangle mesh. FIG. 5 illustrates an example of atriangle mesh formed from the structured light points. For example,triangle 510 is formed from three structured light points 511, 512 and513. Triangle 520 is formed from three structured light points 521, 522and 523. The triangle mesh is formed similar to Delaunay triangulation.Each of the triangles formed is a flat plane surface intersecting withthe three points that comprise the triangles vertices. The plane can befound by interpolating the three depth values.

The normal intensity/color image may show a feature, which can beencircled by a user or a software tool. The algorithm below illustratesan example used to calculate the silhouette area or shadow area of thisfeature.

A free hand loop 6610 to indicate a feature can be projected onto thetriangle mesh as shown in FIG. 6. The area component from each trianglecan be found by simply determining the 2D intersection area of thetriangle and the projected free hand loop.

A triangle (i.e., 520) across the free hand loop as shown in FIG. 7Acontributes partially to the area size estimation. For triangle 520,only the portion (i.e., 710) inside the free hand loop 610 is countedfor area calculation. The portion inside the free hand loop 610 isindicated by the hatch-line filled area 710 in FIG. 7A. A triangle (e.g.triangle 510) may also be entirely in the interior of the free handloop. In this case, the triangle contributes the entire area to the areasize calculation. On the other hand, a triangle may be on the exteriorof the free hand loop and does not contribute to the area size at all.In practice we normally subdivide mesh to denser and smoother mesh. It'seasy to compute the interior area of the free hand drew contour based onmore detailed mesh.

We can compute surface area or cross section area as shown in FIG. 7B.A_(i) is surface area, n_(i) is surface normal, A_(eff) is the crosssection area by projecting the mesh to a plane whose normaln=Σ(n_(i))/∥(n_(i))∥

A projected (cross section) area A_(eff) of the tumor can be defined as:A _(eff)=Σ_(i) ^(K) A _(i)*({circumflex over (n)} _(i) ·n)

In yet another implementation, a surface area A_(surface) can bedetermined from:

$A_{surface} = {\sum\limits_{i}^{K}A_{i}}$

The area derived as above is a better measurement of the size of a tumorthan the simply linear (i.e., 1D) dimension for indicating diseasestate.

In yet another embodiment, the feature is assumed to fit a shape (e.g.ellipsoid, ovoid, sphere, disc or more complicate 3D shapes.).Therefore, the volume for the feature can be estimate from the shapeassumption and parameter(s) associated with the shape using the 3D modelof the object surface.

FIG. 8 illustrates an exemplary capsule system with on-board storage.The capsule device 850 includes illuminating system 812 and a camerathat includes optical system 815 and image sensor 816. A semiconductornonvolatile archival memory 820 may be provided to allow the images tobe stored and later retrieved at a docking station outside the body,after the capsule is recovered. Capsule device 850 includes batterypower supply 825 and an output port 826. Capsule device 850 may bepropelled through the gastrointestinal (GI) tract by peristalsis.

Illuminating system 812 may be implemented by LEDs. In FIG. 8, the LEDsare located adjacent to the camera's aperture, although otherconfigurations are possible. The Illuminating light source may also beprovided, for example, behind the aperture. Other Illuminating lightsources, such as laser diodes, may also be used. Alternatively, whitelight sources or a combination of two or more narrow-wavelength-bandsources may also be used. White LEDs are available that may include ablue LED or a violet LED, along with phosphorescent materials that areexcited by the LED light to emit light at longer wavelengths. Theportion of capsule housing 10 that allows light to pass through may bemade from bio-compatible glass or polymer.

Optical system 815, which may include multiple refractive, diffractive,or reflective lens elements, provides an image of the lumen walls (800)on image sensor 816. Image sensor 816 may be provided by charged-coupleddevices (CCD) or complementary metal-oxide-semiconductor (CMOS) typedevices that convert the received light intensities into correspondingelectrical signals. Image sensor 16 may have a monochromatic response orinclude a color filter array such that a color image may be captured(e.g. using the RGB or CYM representations). The analog signals fromimage sensor 816 are preferably converted into digital form to allowprocessing in digital form. Such conversion may be accomplished using ananalog-to-digital (A/D) converter, which may be provided inside thesensor (as in the current case), or in another portion inside capsulehousing 810. The A/D unit may be provided between image sensor 16 andthe rest of the system. LEDs in illuminating system 812 are synchronizedwith the operations of image sensor 816. Processing module 822 may beused to provide processing required for the system such as imageprocessing and video compression. The processing module may also provideneeded system control such as to control the LEDs during image captureoperation. The processing module may also be responsible for otherfunctions such as managing image capture and coordinating imageretrieval.

After the capsule camera traveled through the GI tract and exits fromthe body, the capsule camera is retrieved and the images stored in thearchival memory are read out through the output port. The receivedimages are usually transferred to a base station for processing and fora diagnostician to examine. The accuracy as well as efficiency ofdiagnostics is most important. A diagnostician is expected to examinethe images and correctly identify any anomaly. While the on-boardarchival memory is used for storing the captured images, a wirelesstransmitter can be used to send to captured images to a receiver locatedout the human body.

In order to capture structured light images, the capsule camera needs aprojection system (not shown in FIG. 8) to project the light beams witha known pattern. The projection system comprises a projection lightsource (e.g. a laser or LED) and an optical system. The same imagesensor 816 can be used to capture the structured light image. Theprocessing module 822 or a separate processor can be used to perform thecomputations required to derive the distance information at various beamlocations. When a table is used to store pre-computed 3D points atvarious target distances, a storage device (e.g. NVRAM (non-volatilerandom access memory) or ROM (read-only memory)) can be used. Thearchival memory may also be used to store the table.

FIG. 9 illustrates an exemplary flowchart for estimating or measuring aphysical length, physical area or physical volume of an object ofinterest in one or more images captured using an endoscope according toan embodiment of the present invention. The steps shown in theflowchart, as well as other following flowcharts in this disclosure, maybe implemented as program codes executable on one or more processors(e.g., one or more CPUs) at the encoder side and/or the decoder side.The steps shown in the flowchart may also be implemented based hardwaresuch as one or more electronic devices or processors arranged to performthe steps in the flowchart. One or more structured-light images arereceived in step 910, wherein said one or more structured-light imagesare captured using the imaging apparatus by projecting structured lightonto a body lumen when the imaging apparatus is in the body lumen. Saidone or more regular images are received in step 920, wherein said one ormore regular images are captured using the imaging apparatus byprojecting non-structured light onto the body lumen when the imagingapparatus is inside the body lumen. The object of interest in said oneor more regular images are determined in step 930, wherein the object ofinterest corresponds to a target object on a wall of the body lumen.Distance information associated with the object of interest with respectto the imaging apparatus is derived from said one or morestructured-light images in step 940. Physical area size or physicalvolume size of the object of interest is determined based on said one ormore regular images and the distance information in step 950. Theimaging apparatus can be a capsule endoscope or an insertion endoscope.

The above description is presented to enable a person of ordinary skillin the art to practice the present invention as provided in the contextof a particular application and its requirements. Various modificationsto the described embodiments will be apparent to those with skill in theart, and the general principles defined herein may be applied to otherembodiments. Therefore, the present invention is not intended to belimited to the particular embodiments shown and described, but is to beaccorded the widest scope consistent with the principles and novelfeatures herein disclosed. In the above detailed description, variousspecific details are illustrated in order to provide a thoroughunderstanding of the present invention. Nevertheless, it will beunderstood by those skilled in the art that the present invention may bepracticed.

The invention may be embodied in other specific forms without departingfrom its spirit or essential characteristics. The described examples areto be considered in all respects only as illustrative and notrestrictive. The scope of the invention is, therefore, indicated by theappended claims rather than by the foregoing description. All changeswhich come within the meaning and range of equivalency of the claims areto be embraced within their scope.

The invention claimed is:
 1. A method of estimating or measuring aphysical area or physical volume of an object of interest in one or moreregular images captured using an imaging apparatus, the methodcomprising: receiving one or more structured-light images, wherein saidone or more structured-light images are captured using the imagingapparatus by projecting structured light onto a body lumen when theimaging apparatus is in the body lumen; receiving said one or moreregular images, wherein said one or more regular images are capturedusing the imaging apparatus by projecting non-structured light onto thebody lumen when the imaging apparatus is inside the body lumen;determining the object of interest in said one or more regular images,wherein the object of interest corresponds to a target object on a wallof the body lumen; deriving distance information associated with theobject of interest with respect to the imaging apparatus from said oneor more structured-light images; and determining physical area size orphysical volume size of the object of interest based on said one or moreregular images and the distance information.
 2. The method of claim 1said determining the physical area size or the physical volume size ofthe object of interest comprises: determining a first direction alignedwith a longest straight line from one end of the object of interest toanother end of the object of interest; estimating a first physicaldimension of the object of interest based on the longest straight linemeasured from said one or more regular images scaled by a magnificationfactor associated with optical configuration of an image sensor of theimaging apparatus; and estimating a second physical dimension of theobject of interest based on a second measured longest length of theobject of interest in a second direction scaled by the magnificationfactor, wherein the second direction is perpendicular to the firstdirection; and wherein the physical area size of the object of interestis proportional to a product of the first physical dimension of theobject of interest and the second physical dimension of the object ofinterest.
 3. The method of claim 1, wherein said determining the objectof interest comprising outlining a boundary of the object of interest bya user or an artificial intelligence process or jointly by the user andthe artificial intelligence process.
 4. The method of claim 3, whereinsaid outlining the boundary of the object of interest is performed bythe user using an input device to indicate the boundary of the object ofinterest in said one or more regular images displayed on a displaydevice.
 5. The method of claim 3, wherein if the object of interestcrosses the boundary of a current regular image into one or moreneighboring regular images, the current regular image and said one ormore neighboring regular images are stitched prior to said determiningthe physical area size or the physical volume size of the object ofinterest.
 6. The method of claim 5, wherein the object of interest isoutlined in the current regular image and said one or more neighboringregular images to assist stitching the current regular image and saidone or more neighboring images.
 7. The method of claim 1, wherein saiddetermining the object of interest comprises outlining a boundary of theobject of interest automatically using image processing.
 8. The methodof claim 1, wherein said determining the physical area size or thephysical volume size of the object of interest is performedautomatically using computer executable codes executed on a computingdevice.
 9. The method of claim 1, wherein the object of interestcorresponds to a lesion, pedunculated polyp, sessile serrated polyp,flat lesion or an infected area by Crohn's disease.
 10. The method ofclaim 1, wherein a score board is generated for an anomaly as anindication of condition of the anomaly by using an aggregate numbercorresponding to an average area or percentage of one or more targetobjects of interest in one or more first regular images representing asection of GI (gastrointestinal) tract, and wherein said one or moretarget objects of interest correspond to one or more infected areas inthe GI tract associated with the anomaly.
 11. The method of claim 10,wherein the section of GI tract corresponds to a part of small bowel, apart of colon or both.
 12. The method of claim 1, wherein an indexnumber is generated for an anomaly as an indication of a disease statefor the anomaly, and wherein the index number is proportional to a sumof infected areas in said one or more regular images in a GI(gastrointestinal) section divided by a sum of total image areas in saidone or more regular images in the GI section.
 13. The method of claim 1,wherein the object of interest corresponds to an anomaly and thephysical volume size of the object of interest is derived based on a 3Dmathematical model for the anomaly and measured 1D or 2D sizeinformation for the object of interest, and wherein the 3D mathematicalmodel is used to predict depth or volume of the anomaly underneath amucosal surface.
 14. The method of claim 13, wherein the 3D mathematicalmodel corresponds to an ellipsoid, ovoid, sphere or disc.
 15. The methodof claim 1, wherein the physical area size or the physical volume sizeof the object of interest is derived based on the distance information,and wherein the distance information is derived at a set of pointlocations by projecting the structured light onto the body lumen. 16.The method of claim 15, wherein a triangle mesh is generated to cover atopographic surface of the object of interest, and wherein the trianglemesh consists of a set of triangles generated using the set of pointlocations.
 17. The method of claim 16, wherein an outline identifyingthe object of interest is projected onto the triangle mesh and projectedarea of a target set of triangles within the outline projected isdetermined as the physical area size of the object of interest.
 18. Themethod of claim 17, wherein a vector sum of target triangles of thetarget set of triangles is calculated, and wherein each vector areaassociated with each target triangle corresponds to a product of areaand normal vector of each target triangle.
 19. The method of claim 16,wherein a 3D shape model is assumed for the object of interest and thephysical volume size of the object of interest is estimated based on ameasured 1D or 2D size information for the object of interest.
 20. Themethod of claim 1, wherein said determining the object of interest insaid one or more regular images comprises indicating the object ofinterest by a user via a computer input device or by executing programcodes on a computer.
 21. The method of claim 20, wherein said indicatingthe object of interest comprises generating an outline around the objectof interest or changing a color or shade of the object of interest. 22.The method of claim 21, wherein the program codes include routines toutilize deep learning or artificial intelligence to automaticallygenerate the outline around the object of interest.
 23. The method ofclaim 1, wherein the imaging apparatus corresponds to a capsuleendoscope.
 24. The method of claim 1, wherein the imaging apparatuscorresponds to an insertion endoscope.
 25. An apparatus for estimatingor measuring a physical area or physical volume of an object of interestin one or more images captured using an imaging apparatus, the apparatuscomprising one or more electronic circuits or processors arranged to:receive one or more structured-light images, wherein said one or morestructured-light images are captured using the imaging apparatus byprojecting structured light onto a body lumen when the imaging apparatusis in the body lumen; receive one or more regular images, wherein saidone or more regular images are captured using the imaging apparatus byprojecting non-structured light onto the body lumen when the imagingapparatus is inside the body lumen; determine the object of interest insaid one or more regular images, wherein the object of interestcorresponds to a target object on a wall of the body lumen; derivedistance information associated with the object of interest with respectto the imaging apparatus from said one or more structured-light images;and determine physical area size or physical volume size of the objectof interest based on said one or more regular images and the distanceinformation.